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In this paper we consider Iq regularized convex cone programming problems. In par- 
ticular, we first propose an iterative hard thresholding (IHT) method and its variant for 
solving Iq regularized box constrained convex programming. We show that the sequence 
generated by these methods converges to a local minimizer. Also, we establish the it- 
^ ' eration complexity of the IHT method for finding an e-local-optimal solution. We then 

propose a method for solving Iq regularized convex cone programming by applying the 
IHT method to its quadratic penalty relaxation and establish its iteration complexity for 
CN I finding an e-approximate local minimizer. Finally, we propose a variant of this method 

in which the associated penalty parameter is dynamically updated, and show that every 
If^ , accumulation point is a local minimizer of the problem. 
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1 Introduction 

Sparse approximations have over the last decade gained a great deal of popularity in numerous 
areas. For example, in compressed sensing, a large sparse signal is decoded by finding a sparse 
solution to a system of linear equalities and/or inequalities. Our particular interest of this 
paper is to find a sparse approximation to a convex cone programming problem in the form 
of 

min f{x) 

s.t. Ax-belC\ (1) 
I < X < u 
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for some / G J?", m G 3?", A G S?'"^"' and b G 3?*", where JC* denotes the dual cone of a closed 
convex cone /C C gf^, i.e., /C* = {s G 3?" : s^x > 0,Vx G /C}, and fJl = {x : -oo < < 
0, 1 < i < n} and 3?" = {x G 3?" : < Xj < cxd, 1 < i < n}. A sparse solution to ([T]) can be 
sought by solving the following Zq regularized convex cone programming problem: 

min /(x) + A||x||o 

s.t. Ax-beIC*, (2) 
I < X < u 

for some A > 0, where ||x||o denotes the cardinality of x. One special case of ([2]), that is, the Iq- 
regularized unconstrained least squares problem, has been well studied in the literature (e.g., 
|13[ [To]), and some methods were developed for solving it. For example, the iterative hard 
thresholding (IHT) methods [HI Ej [3] and matching pursuit algorithms [HI [Tl] were proposed 
to solve this type of problems. Recently, Lu and Zhang [10] proposed a penalty decomposition 
method for solving a more general class of Iq minimization problems. 

As shown by the extensive experiments in [2l |3], the IHT method performs very well in 
finding a sparse solution to unconstrained least squares problems. In addition, the similar 
type of methods [3 E] were successfully applied to find low rank solutions in the context of 
matrix completion. Inspired by these works, in this paper we study IHT methods for solving 
Iq regularized convex cone programming problem (|2]). In particular, we first propose an IHT 
method and its variant for solving Iq regularized box constrained convex programming. We 
show that the sequence generated by these methods converges to a local minimizer. Also, we 
establish the iteration complexity of the IHT method for finding an e-local-optimal solution. 
We then propose a method for solving Iq regularized convex cone programming by applying 
the IHT method to its quadratic penalty relaxation and establish its iteration complexity for 
finding an e-approximate local minimizer of the problem. We also propose a variant of the 
method in which the associated penalty parameter is dynamically updated, and show that 
every accumulation point is a local minimizer of the problem. 

The outline of this paper is as follows. In Subsection 11.11 we introduce some notations 
that are used in the paper. In Section [2] we present some technical results about a projected 
gradient method for convex programming. In Section |3] we propose IHT methods for solving 
Iq regularized box constrained convex programming and study their convergence. In section |4] 
we develop IHT methods for solving Iq regularized convex cone programming and study their 
convergence. Finally, in Section \5\ we present some concluding remarks. 

1.1 Notation 

Given a nonempty closed convex i7 C 3fJ" and an arbitrary point x G N'nix) denotes the 
normal cone of Q at x. In addition, d^ly) denotes the Euclidean distance between y G 9?" and 
fl. All norms used in the paper are Euclidean norm denoted by || ■ ||. We use U{r) to denote 
a ball centered at the origin with a radius r > 0, that is, U{r) := {x G 3?" : ||x|| < r}. 
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2 Technical preliminaries 

In this section we present some technical results about a projected gradient method for convex 
programming that will be subsequently used in this paper. 
Consider the convex programming problem 

0*:=min0(x), (3) 

where X C 3?" is a closed convex set and : X — )■ 3? is a smooth convex function whose 
gradient is Lipschitz continuous with constant > 0. Assume that the set of optimal 
solutions of Q, denoted by X*, is nonempty. 

Let L > he arbitrarily given. A projected gradient of at any x G X with respect to 
X is defined as 

gix) ■.= L[x- Ux (x - V0(x)/L)] , (4) 

where Ilx{-) is the projection map onto X (see, for example, \T2\). 

The following properties of the projected gradient are essentially shown in Proposition 3 
and Lemma 4 of [9] (see also |12j). 

Lemma 2.1 Let x G X 6e given and define x"*" := Ilx{x — V0(x)/L). Then, for any given 
e > 0, the following statements hold: 

a) \\g{x)\\ < e if and only ifV(f){x) G -Afx{x^) + U{t). 

h) \\g{x)\\ < e implies that V0(x+) G -A/'x(a;+) +W(2e). 

c) 0(x+)-0(x)<-||(7(x)||V(2L). 

d) (j){x) — 0(x*) > ||5'(x)|p/ (2L), where x* G Argmin{0(y) : y G X}. 

We next study a projected gradient method for solving 

Projected gradient method for ([3]): 

Choose an arbitrary x° G X. Set = 0. 

1) Solve the subproblem 

x^+^ = argmin{0(x^) + V0(x^)'^(x - x'') + —\\x - x^||^}. (5) 
xgx 2 

2) Set k k + 1 and go to step 1). 
end 

Some properties of the above projected gradient method are established in the following 
two theorems, which will be used in the subsequent sections of this paper. 
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Theorem 2.2 Let {x^} be generated by the above projected gradient method. Then the fol- 
lowing statements hold: 

(i) For every k > and I > 1, 

-0* < ^llx'^ -x*f . (6) 

(ii) {a;^} converges to some optimal solution x* of ([3]). 

Proof, (i) Since the objective function of is strongly convex with modulus L, it follows 
that for every x E X, 

(f){x'')+V(j){x''f{x-x'')+hx-x''f > 0(x'=)+V0(a;'=)^(x'=+^-x^)+^||x^+^-x'=f+^||x-x'=+^ 

By the convexity of 0, Lipschitz continuity of V0 and L > L^, we have 

> (pi^x'') + W (p^x^Y {x - x^) , 
< + V (t){x^'f {x^+^ - x^) + fllx^+i - x^'f , 

which together with the above inequality imply that 

0(x) + ^||x-x'=f > + VxgX (7) 

Letting x = x*^ in ([7]), we obtain that 

0(x'=) - 0(x'=+^) > L\\x^+^ -x^f/2. 
Hence, is decreasing. Letting x = x* E X* in ([7j), we have 

Using this inequality and the monotonicity of {0(x*'')}, we obtain that 

k+l-l J 

l{ct>{x'+^) - ct>*) < ^ - 0*] < _(||a;'^-a;*|p-||a;'=+'-a;*f), (8) 

i=k 

which immediately yields ([6]). 
(ii) It follows from ([8]) that 

\\x^+^-x*\\ < \\x''-x*\\, VA;>0,/>L (9) 

Hence, \\x^ < \\x^ for every k. It implies that {x^} is bounded. Then, there exists 

a subsequence K such that {x^}k — ?■ x* G X. It can be seen from IQ that {(j){x'')}K (p* ■ 
Hence, 0(x*) = \im.ki^K^oo(p{x^) = 4'*, which implies that x* G X*. Since holds for any 
X* G X*, we also have \\xk+i — < \\x'^ — x*\\ for every A; > and I > 1. This together 
with the fact {x''}k x* implies that {x'^'} — )■ x* and hence statement (ii) holds. ■ 
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Theorem 2.3 Suppose that cf) is strongly convex with modulus a > 0. Let {x^} he generated 
by the above projected gradient method. Then, for any given e > 0, the following statements 
hold: 

(i) (h(x^) — 6* < e whenever 



k > 2\L/a] 



log 



(x^) — (/)*< e whenever 



k>2 \L/a] 



l0£ 



ix" - 



+ 1. 



Proof (i) Let M = \L/a]. It follows from Theorem O that 

cPix'^') -<p*<h^'-x*r< -Mx') - 0*), 

ZL aL 
where x* is the optimal solution of (j3]). Hence, we have 

which implies that 

Let K = [log((0(x°) - 0*)/e)]. Hence, when k > 2KM, we have 

which immediately implies that statement (i) holds. 

(ii) Let K and M be defined as above. If (f){x'^^^^) = 0*, by monotonicity of {(j){x^)} we 
have (f){x^) = 0* when k > 2KM, and hence the conclusion holds. We now suppose that 
0^^2XM^ > 0*. It implies that g{x'^^'^^) ^ 0, where g is defined in (jlj). Using this relation. 
Lemma [2.11 (c) and statement (i), we obtain that 0(x^^^"'"^) < 0(x^^*^) < e, which together 
with the montonicity of {0(x'^)} implies that the conclusion holds. ■ 

Finally, we consider the convex programming problem 

/* := min{/(x) : Ax - b e IC\x e X}, (10) 

for some A G S?"*^" and b G 3?"*, where / : X — 3ft is a smooth convex function whose gradient 
is Lipschitz continuous gradient with constant L/ > 0, X C 3ft" is a closed convex set, and K.* 
is the dual cone of a closed convex cone /C. 
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The Lagrangian dual function associated with (fTOj) is given by 

c?(/i) := inf{/(x) + - 6) : x G X}, V^u G -/C. 

Assume that there exists a Lagrange multipher for (ITO!) . that is, a vector /i* G — /C such 
that = /*. Under this assumption, the following results are established in Corollary 2 

and Proposition 10 of [9], respectively. 

Lemma 2.4 Let n* be a Lagrange multiplier for ( ITOl) . There holds: 

fix)- r >-\\fi*\\d,C'iAx-b), yxex. 
Lemma 2.5 Let p > be given and Lp = Lf + p||y4|p. Consider the problem 

:= min{$,(x) := /(x) + ^[d^^^x - b)]'}. (11) 

If X & X is a C,- approximate solution of (ITT]) , z.e., $p(a;) — $p < ^, then the pair {x^,fi) 
defined as 

x+ := nx(x- V$p(x)/L,), 
/i := p[Ax+ -b-U!c* {Ax+ - b)] 
is in X X (— /C) and satisfies fi^Ilic*{Ax^ — b) = and the relations 



dK.*{A^+-b) < i||p*|| + ^S, 

Vf{x+) + A^p G -Mx{x^)+U{2^2L;i), 

where p* is an arbitrary Lagrange multiplier for (fTO|) . 

3 /o regularized box constrained convex programming 

In this section we consider a special case of ([2]), that is, /q regularized box constrained convex 
programming problem in the form of: 

F*:=min := /(x) + A||x||o 

s.t. I < X < u 

for some A > 0, / G and u G Recently, Blumensath and Davies [21 [3] proposed 

an iterative hard thresholding (IHT) method for solving a special case of ( fT2l) with /(x) = 
1 1 Ax — h = —oc and = oo for all i. Our aim is to extend their IHT method to solve 
(fT2|) and study its convergence. In addition, we establish its iteration complexity for finding 
an e-local-optimal solution of (fT2|) . Finally, we propose a variant of the IHT method in which 
only "local" Lipschitz constant of V/ is used. 
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Throughout this section we assume that / is a smooth convex function in B whose gradient 
is Lipschitz continuous with constant Lf > 0, and also that / is bounded below on the set B, 
where 

B := {x eW" -.1 <x <u}. (13) 
We now present an IHT method for solving problem ( fT2|) . 

Iterative hard thresholding method for (1121): 

Choose an arbitrary G B. Set k = 0. 

1) Solve the subproblem 

^k+i ^ Argmin{/(x^) + V/(x^)^(x - x'') + -\\x - x'^f + \\\x\\o}. (14) 

x£B 2 

2) Set /c A; + 1 and go to step 1). 
end 

Remark. The subproblem f lT4|) has a closed form solution given in fl2T]) . ■ 

In what follows, we study the convergence of the above IHT method for flT2|) . Before 
proceeding, we introduce some notations that will be used subsequently. Define 

Bi := {x eB : XI = 0}, V/ C {1, . . . , n}, (15) 

nB(x) := aigm.m{\\y — x\\ : y e B}, Vx G 3?", 

Sl{x) := x-^Vf{x), \fxeB, (16) 

I{x) := {i:Xi = 0}, Vx G 3ft" (17) 

for some constant L > Lf. 

The following lemma establishes some properties of the operators sl{-) and IIs{sl{-)), 
which will be used subsequently. 

Lemma 3.1 For any x, y E 3ft", there hold: 

(1) \[sL{x)]j - [s^iym < 4(||x - y\\ + |[.l(i/)].|)||x - y\\; 

(2) |[nB(sL(x))-SL(x)]f-[ns(sL(y))-sz.(y)]f| <4(||x-y|| + |[nB(sL(y))-sz.(y)].|)||x-y||. 

Proof. (1) We observe that 

||sz.(x)-Si(i/)|| = ||x-y-i(V/(x)-V/(y))|| < ||x-i/|| + i||V/(x)-V/(i/)||, 

< (l + :^)||a;-y|| < 2\\x-y\\. (18) 
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It follows from 1^ that 

\[sL{x)]^-[sL{ym = \[sL{x)h + [sL{y)U-\[sL{x)].-[sL{y)]l 

< i\[sL{x)]^ - [sLivM + 2\[sLiym ■ \[sLix)], - [sL{y)]^ 

< A{\\x-y\\ + \[s,iy)mx-yl 

(2) It can be shown that 

\\Ilt3{x)-x + y-Ili3iy)\\ < \\x-y\\. 
Using this inequality and (fTSl) . we then have 

msisLix)) - SLix)]^ - [nsisM) - s^iym 

< msisLix)) - SLix)], - [IlsisLiy)) - sMU + 2\Us{sL{y)) - SL{yM 
■mBi-SLix)) - si^{x% - [ne(si(2/)) - si^{y)\l 

< i\\s,{x) - s,iy)\\ + ms{s,{y)) - s.iyM ■ \\s,ix) - s,iy)l 
<Ai\\x-y\\ + \[UsisL{y))-sUym\\x-y\\. 



The following lemma shows that for the sequence {x'^}, the magnitude of any nonzero 
component cannot be too small for A; > 1. 

Lemma 3.2 Let {x'^} be generated by the above IHT method. Then, for all k > 0, 

> 6:=mm6i > 0, z/ xJ+V 0, (19) 

where Iq = {i : li = Ui = 0} and 

{min{ui, a/2A/L), ifk = 0, 
mm{-li,,/2X/L), if Ui = 0, Vz G Jq. (20) 

min(— a/2A/L), otherwise, 

Proof. One can observe from f lT^ that for i = 1, . . . ,n, 

r [YlsisLix'm, if [sl{x')]1 - [UsMx")) - sl{x^)]] > f, 

x1^' = I 0, if [sl{x>^)]^ - [U^isLix')) - sl{x')]^ < (21) 

[ [ng(si(x'''))]j or 0, otherwise 

(see, for example, [ID]). Suppose that j is an index such that x^^^ ^ 0. Clearly, j ^ Iq, where 
If) is define above. It follows from fl2T]) that 

2A 

= [ns(.,(a:^))], ^ 0, Mx'^)]^ - [Usis.ix'^)) - s.ix'^)]'^ > -. (22) 
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The second relation of (122|) implies that |[sL(a;'^)]j| > ^/iXjL. In addition, by the first relation 
of (I22!) and the definition of Ilg, we have 



xf^ = [Us{sl{x''))], = mm{max{[sL{x%,lj),u,) ^ 0. (23) 

Recall that j ^ /q. We next show that |a;^^^| > bj by considering three separate cases: i) 
Ij = 0; ii) Uj = 0; and iii) IjUj 7^ 0. For case i), it follows from ( l23l) that [sL{x'')]j > 
and x^~^^ = mm([sL{x'')]j,Uj). This together with the relation |[sL(x'')]j| > a/2A/L and 
the definition of 5j implies that l^^j"*"^! > Sj. By the similar arguments, we can show that 
la;^"^^! > 6j also holds for the other two cases. Then, it is easy to see that the conclusion of 
this lemma holds. ■ 

We next establish that the sequence {x^} converges to a local minimizer of f|T2|) . and 
moreover, F{x^) converges to a local minimum value of (1121) . 

Theorem 3.3 Let {x^} be generated by the above IHT method. Then, x^ converges to a local 
minimizer X* of problem (fT2|) and moreover, I{x^) — > I{x*), \\x^\\q \\x*\\o and F{x^) — t- 
F{x*). 

Proof. Since V/ is Lipschitz continuous with constant Lf, we have 

fix''^^) < f{x'') + Vf{x''f{x~x'') + ^\\x''+^-x''f. 
Using this inequality, the fact that L > Lf, and (JT4l) . we obtain that 



F(x^+i) = /(x'^+i) + Allx'^+io < fix'') + Vfix''f{x''+^-x^) + ^\\x''-^'-x^f + X\\x^+^\ 
< fix') + Vfix'fix'-^' - x') + ^\\x'+' ~x'f + A||x^+io 



b 

< fix'') + X\\x^\\o = F(x^), 



where the last inequality follows from f|T4|) . The above inequality implies that {F(x'^)} is 
nonincreasing and moreover, 

F(x'=) - F(x'=+^) > b-a = -x^f . (24) 

By the assumption, we know that / is bounded below in B. It then follows that {F{x'')} is 
bounded below. Hence, {F{x'')} converges to a finite value as — ?■ 00, which together with 
f l2il) implies that 

lim ||a;'=+^ -x'^ll = 0. (25) 



Let Ik = I{x^), where /(■) is defined in (fT71) . In view of (fT9|) . we observe that 

\\x^+^-x%>5 if 4^4+1. (26) 

This together with fl25l) imphes that 4 does not change when k is sufficient large. Hence, 
there exist some K > and / C {1, . . . , n} such that Ik = I for all k > K. Then one can 
observe from (flU) that 



= argmin{/(x'=) + Vfix'' f{x - x^) + -\\x - x^f}, Mk > K, 

xeBi 2 

where Bj is defined in f lT5]) . It follows from Lemma [2.21 that x^ ^ x*, where 

X* e Aigmm{f{x) : X e Bi}. (27) 

It is not hard to see from f l27p that x* is a local minimizer of f ll2p . In addition, we know from 
( IT^ that > 5 for A; > and i ^ /. It yields \x*\ > 6 for i ^ I and x* = for i G /. 
Hence, I{x^) = I{x*) = I for all k > K, which clearly implies that Hx'^'Ho = ||a^*||o for every 
k > K. By continuity of /, we have /(a;*^) — )■ f{x*). It then follows that 

F(x'=) = fix'') + X\\x''\\o fix*) + X\\x*\\o = Fix*). 



As shown in Theorem l3.3[ x'' — )■ x* for some local minimizer x* of fll2p and Fix'') — )■ Fix*). 
Our next aim is to establish the iteration complexity of the IHT method for finding an e-local- 
optimal solution x^ & B of f lT2|) satisfying Fixe) < Fix*) + e and /(x^) = /(x*). Before 
proceeding, we define 



a = mm 

/C{l,...,n} 



mm 



[sLix*)]^ - mSLix*)) - S,ix*)]^ - - 



: X* E Argmin{/(x) : x E i3/|28,) 



/3 = max \meLx\[sLix*)]i\ + lUjsisLix*)) - SLix*)]i\ : x* G Argmin{/(x) : x E Bi}\ . (29) 

/C{l,...,n} I i J 

Theorem 3.4 Assume that f is a smooth strongly convex function with modulus a > 0. 
Suppose that L > Lf is chosen such that a > 0. Let {x''} be generated by the above IHT 
method, I^ = /(x^) for all k, x* = limfc_!.oo x'^ , and F* = Fix*). Then, for any given e > 0, 
the following statements hold: 

(i) The number changes of 1^ is at most 



2{F{x°)-F*) 
{L-Lf)S^ 



(ii) The total number of iterations by the IHT method for finding an e- local- optimal solution 
x^ E B satisfying /(x^) = /(x*) and Fix^) < F* + e is at most 2[L/cr] log where 



e 



(F(x^)-F*)2 



bj+3 



max I (d - 2c)t -ct"^ -.0 <t< 



2(F(a:")-F*) 
(L-Lf)S2 



^InJ^Cr) ^ 7 = ^(v/2^T^-/3)V32, 
ci = 2 log(F(x°) -F*) + 4-21og7 + c. 



, (30) 
(31) 
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Proof, (i) As shown in Theorem 13. 3[ Ik only changes for a finite number of times. Assume 
that Ik only changes at k = rii + 1, . . . ,nj + 1, that is, 

Irij.i+l = ■ ■ ■ = Irij ^ ^rij + l = ■ ■ ■ = luj+i, j = ^, ■ ■ ■ , J — ^, (32) 

where Uq = 0. 

We next bound J, i.e., the total number of changes of Ik- In view of ( 126|) and ( 132|) . one 
can observe that 

j = l,...,J, 

which together with (|2^ implies that 



F(x"0-i^(a:"^+')>^(^-^/)5', J = 1,...,J. (33) 
Summing up these inequalities and using the monotonicity of {F(x'^)}, we have 

i(L-L/)5V < F(x"^) -F(x"'^+i) < F(x°)-F*, (34) 

and hence 



J < 



2{F{x^) - F*) 



(L - Lf)6' 

(ii) Let rij be defined as above for j = 1, . . . , J. We first show that 



(35) 



nj-n,_, < 2 + 2\L/a]\\og{F{x')~{j-l){L-Lf)6y2-F*)~\og^], j = 1, . . . , J, 

(36) 

where F_* and 7 are defined in f[T^ and flHTl) . respectively. Indeed, one can observe from f[T^ 
that 

x^+i = argmin{/(x'') + Vf{x^Y{x - x'') + — ||a; - x^\\'^ : Xj = 0}. 
xeB 2 

Therefore, for j = 1, . . . , J and k = rij^i, . . . ,nj — 1, 

x'^+i = argmin{/(x'') + V f{x^Y{x - x^) + —\\x- x^\\^ : a;/„. = 0}. 
xes 2 ^ 

We arbitrarily choose 1 < j < J. Let x* (depending on j) denote the optimal solution of 

min{/(x) : Xi = 0}. (37) 

One can observe that 

IF llo ^ \\x iio- 
Also, it follows from fl33|) and the monotonicity of {F{x'^)} that 

F(a;"^+i) < F(x°)-|(L-L;)52, j = 1, . . . , J. (38) 
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Using these relations and the fact that F{x*) > F_* , we have 



o-F{x*) + X\\x*\\o, 



J-1 



{L-Lf)6^-F*. 



(39) 



Suppose for a contradiction that (136|) does not hold for some 1 < j < J. Hence, we have 
n, - > 2 + 2[L/al [log - (j - 1)(L - Lf)6y2 - F*) - log 7] • 



This inequality and (139|) yields 

- > 2 + 2[L/o-] 



log 



/(a;' 



fix*] 



1 



Using the strong convexity of / and applying Theorem 12.31 (ii) to (|37l) with e = 7, we obtain 
that _ 



2 " 



It implies that 



- a;*|| < 



32 

^2a + 13^-13 



(40) 



Using ( HOl) . Lemma [3. II and the definition of /3, we have 

I[^l(x"0]^? - [sl{x*)]] - [ns(.^(a:"0) - s^ix^^ + [n^(.x(r)) - s,(r)],^| 

< \[sl{x-^)]} - [sl{x*)]1\ + |[ns(.L(x"0) - ^L(a:"0]- - ^^(^.(r)) - s,(r)],^| 

< 4(2||x"^ +/3)||x"^ < a, 
where the last inequality is due to fHDl) . Let 

2A' 



(41) 



z:[.4r)]i=-[nH(.L(r))-.4r)]i=< 



and let J* = {1, . . . , n} \ J*. Since a > 0, we know that 



2A 



[sL{x*)]i - [YiB{sL{x*)) - SL{x*)]i >—, G /*. 



It then follows from ( HTl) and the definition of a that 

[si(x"0]^ - [nB(si(x"0) - sl(x"0]' > f , V2 G 7*. 

Observe that [ng(si(x"-'))]j 7^ for all i G 7*. This fact together with (12T]) implies that 

= 0, z G /* and ^ 0, i G 7*. 
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By a similar argument, one can show that 



0, z G /* and x"^ ^ 0, ieV 



Hence, /„ 



Irij+i = I*, which is a contradiction to ( 132|) . We thus conclude that ( l36l) holds. 
Let denote the total number of iterations for finding an e-local-optimal solution G i3 
by the IHT method satisfying I{x^) = I{x*) and F{x^) < F* + e. We next establish an upper 
bound for A^^^- Summing up the inequality (136|1 for j = 1, . . . , J, we obtain that 



nj< J]<j2 + 2[L/al 



log(F(a;'' 



J-1 



{L-Lf)6'-F*)- log J 



Using this inequality, (!34ll . and the facts that L > a and log(l — t) < —t for all t G (0, 1), we 
have 



nj < 

J 



2 + 2[LH ( log(F(x°) - ^(L - Lf)5^ - F*) - log7 + 1 



2 + 2\L/a] \og{F{x")-F* 



2{F{xO)-F* 



-{j - 1) - log7 + 1 



(^21og(F(a;°) -F*) + 4-21og7 + 



2{F{x^)-F*] 



2{F{x^)-F*) 



By the definition of nj, we observe that after nj + 1 iterations, the IHT method becomes the 
projected gradient method applied to the problem 

X* = argmin{/(a;) : a;/„ = 0}. 

In addition, we know from Theorem 13.31 that /(x^) = I{x*) for all k > rij. Hence, f{x^) — 
f{x*) = F{x^) — F* when k > nj. Using these facts and Theorem 12.31 (ii), we have 



Ne <nj + l + 2\L/a] 



log 



Fix 



^^J+l^ 



Using this inequality, (jSH]), (02]) and the facts that F* > F*, L > a and log(l - t) < -t for 
all t G (0, 1), we obtain that 

< nj + l + 2\L/a](^\og{F{x')-^{L-Lf)6^-F*) + l-\ogey 

< nj+ \L/a] (2 log(F(x°) - F*) - ^t'.^^K'^ + 3 - 2 log e 



- F* 

< \L/a] [(rf - 2c) J-cj2 + 2 log(F(x°)-F*) + 3-21oge] 
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which together with ( 135|) and ( !30|) imphes that 



K < 2\L/a]\og-. 



The iteration complexity given in Theorem 13 .41 is based on the assumption that / is strongly 
convex in B. We next consider a case where B is bounded and / is convex but not strongly 
convex. We will establish the iteration complexity of finding an e-local-optimal solution of 
(fT2|) by the IHT method applied to a perturbation of (fT2!l obtained from adding a "small" 
strongly convex regularization term to /. 

Consider a perturbation of (fT2!) in the form of 

El := min{F,(x) := U{x) + X\\x\\o}, (43) 

where u > and 

:=/(x) + pxf. 

One can easily see that f„ is strongly convex in B with modulus u and moreover V/i/ is 
Lipschitz continuous with constant L^, where 

L, = Lf + u. (44) 



We next establish the iteration complexity of finding an e-local-optimal solution of ( |T2l) 
by the IHT method applied to (H3i) . Given any L > 0, let sl, a and /3 be defined according 
to (fT6|) . (128!) and (!29|l . respectively, by replacing / by /y, and let 5 be defined in (fT9l) . 

Theorem 3.5 Suppose that B is bounded and f is convex but not strongly convex. Let e > 
be arbitrarily given, D = max{||x|| : x G B}, v = e/D'^ , and L > be chosen such that a > 0. 
Let {x''} be generated by the IHT method applied to (143!) . and let x* = limfc_j.oo , F* = F^{x*) 
and F* = min{F(x) : x G Bj*}, where I* = {i : x* = 0}. Then, the total number of iterations 
by the IHT method for finding an e-local-optimal solution x^ E B satisfying F{x^) < F* + e is 



at most 2 



log — , where 



e = {F^{x^) - F*)2^, 00 = max |(d - 2c)t - ct^ : < t < 



2(F.(xO)-FJ) 



^ = 2iMxo]% , 7 = ^(v/2^T^ - /3)V32, 

d = 2 log(F,(a;0) - F^) + 4 - 2 log7 + c. 

Proof. By Theorem 13.41 (ii), we see that the IHT method applied to ( H3l) finds an e/2- 
local-optimal solution G i3 of (H3l) satisfying /(x^) = I{x*) and Ft,{xe) < F* -\- e/2 within 
2\L^/h'~\ log Y iterations. From the proof of Theorem 13.31 we observe that 

F^{x*) = min{F^{x) : x e Bj*}. 
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Hence, we have 

F: = F^ix*) < nnn/(x) + ^ < F* + 

In addition, we observe that F{x^) < Fi,{x^). Hence, it follows that 

Fix,) < F,{x,) < + i < F* + e. 

Note that F* is a local optimal value of (fT2|l . Hence, x, is an e-local-optimal solution of (fT2|) . 
The conclusion of this theorem then follows from (jUj) and u = e/D"^. u 

For the above IHT method, a fixed L is used through all iterations, which may be too 
conservative. To improve its practical performance, we can use "local" L that is update 
dynamically. The resulting variant of the method is presented as follows. 

A variant of IHT method for ( |T2l) : 

Let < Lmin < -^vmaxj T > 1 and 1] > he given. Choose an arbitrary x^ (z B and set A; = 0. 
1) Choose L° G [Lmin, i^max] arbitrarily. Set = L^. 
la) Solve the subproblem 

(z Argmin{/(x*^) + Vf{x''f{x - x^) + ^\\x - x^f + A||x||o}. (45) 



lb) If 



F(x'=) -F(x'=+i) > -x'^f (46) 



is satisfied, then go to step 2). 
Ic) Set Lk ^ rLk and go to step la). 

2) Set /c •(— /c + 1 and go to step 1). 

end 

Remark. can be chosen by the similar scheme as used in [H H], that is, 

TO fr ■ St ^^^^^ 

Li = max < Lmin, mm < L 



where Ax = x^ — x'^^^ and A/ = V/(x'^) — V/(x^^"'^). ■ 

At each iteration, the IHT method solves a single subproblem in step 1). Nevertheless, its 
variant needs to solve a sequence of subproblems. We next show that for each outer iteration, 
its number of inner iterations is finite. 
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Theorem 3.6 For each k > 0, the inner termination criterion (H6|) is satisfied after at most 
iog(L,+,)-iog(L^,„) 2] ,nner iterations. 

Proof. Let denote the final value of at the fcth outer iteration. By f l45|) and the 
similar arguments as for deriving ( 12^ . one can show that 

2 

Hence, ( l46l) holds whenever Lk > Lf + rj, which together with the definition of Lk implies that 
Lk/r < Lf + rj, that is, Lk < T{Lf + rj). Let Uk denote the number of inner iterations for the 
kth outer iteration. Then, we have 



Hence, < 



log(Lj+r?)-log(Lmin) _|_ 2 



and the conclusion holds. 



We next establish that the sequence {x^} generated by the above variant of IHT method 
converges to a local minimizer of (1T2|) and moreover, F{x'^) converges to a local minimum 
value of dn}. 

Theorem 3.7 Let {x^} he generated by the above variant of IHT method. Then, x^ converges 
to a local minimizer X* of problem ( lT2l) . and moreover, I{x^) I{x*), ||a;'^||o \\x*\\o Oj^i^d 
F{x^) F{x*). 

Proof. Let Lk denote the final value of Lk at the A;th outer iteration. From the proof of 
Theorem 13. 6[ we know that Lk € [I/inin,r(L/ + 77)). Using this fact and a similar argument as 
used to prove (fTOll . one can obtain that 



|xf+^| > 5:=mm5i > 0, if xf+V 0, 



where Iq = {i : k = Ui = 0} and 6i is defined according to fl20|) by replacing L by t(L/ + rj) 
for all i E Iq. It implies that 



The conclusion then follows from this inequality and the similar arguments as used in the 
proof of Theorem 13. 3[ ■ 



4 /o-regularized convex cone programming 

In this section we consider /o-regularized convex cone programming problem ([2]) and propose 
IHT methods for solving it. In particular, we apply the IHT method proposed in Section 
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[3] to a quadratic penalty relaxation of ([2]) and establish the iteration complexity for finding 
an e-approximate local minimizer of (|2]). We also propose a variant of the method in which 
the associated penalty parameter is dynamically updated, and show that every accumulation 
point is a local minimizer of (E]). 

Let B be defined in (fT3l) . We assume that / is a smooth convex function in B, V/ is 
Lipschitz continuous with constant Lf and that / is bounded below on B. In addition, we 
make the following assumption throughout this section. 

Assumption 1 For each I C {1, . . . ,n}, there exists a Lagrange multiplier for 

f* = min{/(x) : Ax - b e IC*,x e Bi}, (47) 

provided that (l47l) is feasible, that is, there exists fi* G — /C such that fj = di{fi*), where 

di{fi) := inf{/(x) + f/'iAx - b) : x e Bj}, V/i G -/C. 

Let X* be a point in B, and let I* = {i : x* = 0}. One can observe that x* is a local 
minimizer of ([2]) if and only if x* is a minimizer of ( 1471) with 1 = 1*. Then, in view of 
Assumption [H we see that x* is a local minimizer of ([2]) if and only if x* G i3 and there exists 
11* G —JC such that 

Ax*-belC*, (fi*f(Ax*~b)=0, 

J (48) 
Vf{x*) + A^fi* G -ATb,, (x*). 

Based on the above observation, we can define an approximate local minimizer of ([2]) to 
be the one that nearly satisfies (B8|) . 



Definition 1 Let x* be a point in B, and let I* = {i : x* = 0}. x* is an e-approximate local 
minimizer of ([2]) if there exists fi* G —JC such that 

d,c*iAx* -b) < e, {fx*fUic'{Ax* -b) = 0, 
Vf{x*) + A^^i* G -Mb,, {x*) + U{e). 

In what follows, we propose an IHT method for finding an approximate local minimizer of 
([2]). In particular, we apply the IHT method or its variant to a quadratic penalty relaxation 
of ([2]) which is in the form of 



% := min{^p(x) := $p(x) + A||s||o}, (49) 

where 



$p(x) := f{x) + '^[d^.{Ax-b)f (50) 

It is not hard to show that the function $p is convex differentiable and moreover V$p is 
Lipschitz continuous with constant 

L, = L; + pPf (51) 
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(see, for example, Proposition 8 and Corollary 9 of [9]). Therefore, problem (H9|) can be 
suitably solved by the IHT method or its variant proposed in Section [31 

Under the assumption that / is strongly convex in B, we next establish the iteration 
complexity of the IHT method applied to (l49l) for finding an approximate local minimizer of 
([2]). Given any L > 0, let sl, a and (3 be defined according to (|T6l) . (J28l) and (J29l) . respectively, 
by replacing / by $p, and let 6 be defined in f|T9l) . 

Theorem 4.1 Assume that f is a smooth strongly convex function with modulus a > 0. Given 
any e > 0, let 

P=- + -7^ (52) 

e VS\\A\\ 

for any t > max min where Aj is the set of Lagrange multipliers of f HTl) . Let L > Lp 

7C{l,...,n} AtSAj 

he chosen such that a > 0. Let {x^} he generated hy the IHT method applied to fH^ . and 
let X* = limfc_!.oo a;'^ and \E'* = \E'p(x*). Then the IHT method finds an e-approximate local 
minimizer of (|2]) in at most 



N -.= 2 



a 



, SLpO 
log- 



e2 



iterations, where 

e = {^p{x^) - vl/;)2^, u = max {(d - 2c)t - ct^ : < t < [^^^^^J^J } , 

^ = ^i^'iifnr ^ = ^( v/2^+^ - /3)V32, 

d = 2 log(^p(a;°) - + 4 - 2 log 7 + c. 

Proof. We know from Theorem 13.31 that x'^ — )■ x* for some local minimizer x* of (jH]), 
I{x^) — )■ I{x*) and \l/p(a;^) — )■ \l/p(a;*) = By Theorem 13 .4^ after at most N iterations, the 
IHT method generates x e B such at I{x) = I{x*) and \&p(x) - "^pix*) < ^ := e^/(8Lp). It 
then follows that ^p{x) — $p(x*) < ^. Since x* is a local minimizer of (H9|) . we observe that 

X* = arg min ^p{x), (53) 

where /* = I{x*). Hence, x is a ^-approximate solution of (l53i) . Let /i* G Argmin{||yu|| : 
/i G A/.}, where A/* is the set of Lagrange multipliers of (HTj) with 1 = 1*. In view of 
Lemma 1231 we see that the pair (x+,/i) defined as x"*" := H^^, (x — V$p(x)/Lp) and /i := 
p[Ax~^ — b — Ujc* (Ax"*" — 6)] satisfies 

V/(x+) + A^/i G -A/'s,.(x+)+W(2v/2i;e) =A/e,(5+)+W(e), 

d^.(Ax+-6) < i||/.1l + yf < Kll^*ll + 7fe) ^ ^' 

where the last inequality is due to (1521) and the assumption t > i > Hence, x"*" is an 

e-approximate local minimizer of ([2l). ■ 
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We next consider finding an e-approximate local minimizer of (|2]) for the case where B is 
bounded and / is convex but not strongly convex. In particular, we apply the IHT method 
or its variant to a quadratic penalty relaxation of a perturbation of (E]) obtained from adding 
a "small' ' strongly convex regularization term to /. 

Consider a perturbation of (|2]) in the form of 

min{/(x) + -llxf + A||x||o : Ax-beIC*}. (54) 

x£B 2 

The associated quadratic penalty problem for flM|) is given by 

%,u ■= niin{^p,,(x) := <l>p,,(x) + A||a;||o}, (55) 

where 



$,,.(x) := fix) + pxf + ^[d^^iAx - 6)]2. 

One can easily see that ^p^^ is strongly convex in B with modulus u and moreover V^p^u is 
Lipschitz continuous with constant 

Lp^^ := Lf + p\\Af + u. 

Clearly, the IHT method or its variant can be suitably applied to ( 155|) . We next establish 
the iteration complexity of the IHT method applied to ( l55l) for finding an approximate local 
minimizer of ([2]). Given any L > 0, let s^, a and (3 be defined according to (fT6!) . (!28|) and 
(!29|) . respectively, by replacing / by ^p,u, and let 6 be defined in (fT9l) . 

Theorem 4.2 Suppose that B is bounded and f is convex but not strongly convex. Let e > 
be arbitrarily given, D = max{||x|| : x & B}, 

D + jD + 16t+^^ ' 



p = -^^ " " ^ , u = — (56) 

^ 16e '2D ^ ' 

for any t > max min llulL where Aj is the set of Lagrange multipliers of fHTl) . Let L > L^^ 

/C{l,...,n} AteA/ 

he chosen such that a > 0. Let {x^} be generated by the IHT method applied to fl5^ . and 
let X* = limfc_!.oo a^'^ and '^*p^ = "^p^ui^*)- Then the IHT method finds an e-approximate local 
minimizer of (|2]) in at most 



N -.= 2 

iterations, where 



2DLp,, 



32Lp^9 

log 



e2 



6 = i^pA^') - ^;,.)2^, a; = max |(rf - 2c)t - ct^ : < t < ^^^^0t^ 
d = 2 log(^p,,(xO) - %A + 4 - 2 log7 + c. 
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Proof. From Theorem 13.31 we know that — )■ x* for some local minimizer x* of ([55 



I{x'') — I{x*) and \l>p^,,(x^) — )■ \I'p^i,(x*) = \E'*^. By Theorem 13 ■4[ after at most iterations, 
the IHT method applied to (j55l) generates x E B such at /(a;) = I{x*) and \l/p,,y(x) — \l/p,i,(x*) < 
^ := / {?>2Lp^y). It then follows that $p,i,(x) — $p,i,(x*) < ^. Since x* is a local minimizer of 
fl55l). we see that 



X* = arg min $p j^(a;), (57) 

where /* = I{x*). Hence, x is a ^-approximate solution of ( 1571) . In view of Lemma \2.5\ we 
see that the pair {x~^,fi) defined as x"*" := IlBiix — V^p^^{x)/ Lp^u) and := p[Ax~^ — b — 
Il!C*{Ax~^ — b)] satisfies 

V/(x+) + + e -A/B,,(£+)+W(2v/2Z;;;:e) = -A/'B,.(5+)+W(e/2), 

which together with the fact that z^||x^|| < uD < e/2 implies that 

V/(x+)+AV e -A/'f5,.(£+) +W(e/2) C -ArB^^(5+) +ZY(e). 

In addition, it follows from Lemma [2.11 (c) that $p^,y(x^) < $p.i^(x), and hence 

<l>p,,(£+) - <l>p,,(x*) < <I>p,,(x) - <l>p,,(x*) < e 

Let $p = min{$p(x) : x G Bi*}, where <l>p is defined in (jSU]). Notice that $p,,,(x*) < $* + 
z/L'V2- It then follows that 

$p(x+)-$; < $p,.(5+) - + ^ ^ ^ + T - 32^11' 

Let ;U* G Argmin{||yu|| : p G A/.}, where A/* is the set of Lagrange multipliers of ( H7|) with 
1 = 1*. In view of Lemma [2.51 and the assumption t > t > we obtain that 



where the last inequality is due to ( l56|) . Hence, is an e-approximate local minimizer of (E]). 



For the above method, the fixed penalty parameter p is used through all iterations, which 
may be too conservative. To improve its practical performance, we can update p dynamically. 
The resulting variant of the method is presented as follows. Before proceeding, we define the 
projected gradient of $p at x G Bi with respect to Bj as 

g{x; p, I) = Lp[x - Hb,(x - ^V$p(x))], (58) 

where / C {1, . . . , n}, and $p and Lp are defined in ([50!) and (ISTI) . respectively. 
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A variant of IHT method for ([2]): 

Let {cfc} be a positive decreasing sequence. Let po > 0, t > 1, t > max min ||yu||, where A/ 

/C{l,...,n} AtGAj 

is the set of Lagrange multipliers of fj47|) . Choose an arbitrary x° G B. Set A; = 0. 

1) Start from a;*^"^ and apply the IHT method or its variant to problem fH9|) with p = Pk 
until finding some x'^ & B such that 

< — , ||^(a;^pfc,4)|| < min{l, Lpjefc, (59) 
pk 

where Ik = I{x^). 

2) Set pk+i ■.= Tpk. 

3) Set A; + 1 and go to step 1). 
end 

The following theorem shows that satisfying ( !59l) can be found within a finite number 
of iterations by the IHT method or its variant applied to problem ( l49l) with p = pk- Without 
loss of generality, we consider the IHT method or its variant applied to problem (l49l) with any 
given p > 0. 

Theorem 4.3 Let Xq & B be an arbitrary point and the sequence {xi} be generated by the 
IHT method or its variant applied to problem ( [7^ - Then, the following statements hold: 

(i) lim g{xi; p, Ii) = 0, where Ii = I{xi) for all I. 

(ii) lim dicAAxi — b) < -, where i := max min \\p\\ and A/ is the set of Lagrange multi- 

l^oo P /C{l,...,n} MeA/ 

pliers of (H7|) . 

Proof, (i) It follows from Theorems 13.31 and 13.71 that xi — )■ x* for some local minimizer x* 
of (jH]) and moreover, ^p{xi) $p(x*) and Ii -)■ J*, where J/ = I{xi) and /* = I{x*). We 
also know that 

X* G Arg min $p(x). 
It then follows from Lemma 12.11 (d) that 

$p(xO - <l>p(x*) > ^\\g{xupj*)\\\ 1>N. 

Using this inequality and $p(x;) — )• $p(a;*), we thus have g{xi]p,I*) — )■ 0. Since Ii = I* for 
I > N, we also have g{xi; p, Ii) — )■ 0. 
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(ii) Let // be defined in (H7|) . Applying Lemma [2.41 to problem P7|) . we know that 

f{xi)-f;^i) > -idjc4Axi-b), V/, (60) 

where t is defined above. Let x* and I* be defined in the proof of statement (i). We observe 
that //* > $p(x*). Using this relation and ( I6OII . we have that for sufficiently large /, 

$p(xz) - = f{xi) + l[d,c4Axi-b)f~<l>p{x*) > f{xi)-fJ, + l[d^4Axi-b)]^ 

= fixi)-f*^,^ + l[d,c<Axi-b)f> -idK^{Axi-b) + l[dic^{Axi-b)f, 

which implies that 

p V p 

This inequality together with the fact hm/_s.oo ^p{xi) = $p(x*) yields statement (ii). ■ 



Remark. From Theorem 14. 3[ we can see that the inner iterations of the above method 
terminates finitely. ■ 

We next establish convergence of the outer iterations of the above variant of the IHT 
method for ([2]). In particular, we show that every accumulation point of {x^} is a local 
minimizer of ([2]). 

Theorem 4.4 Let {x'^} be the sequence generated by the above variant of the IHT method for 
solving ([2]). Then any accumulation point of {x''} is a local minimizer of (|2]). 



Proof. Let 



x' = Us,^ix'-^V^,,ix'')). 



Since {x^} satisfies (1591) . it follows from Lemma [2.11 (a) that 

V^p,{x^) e -Aris,^{x')+U{ek), (61) 

where = I{x^). Let x* be any accumulation point of {x^}. Then there exists a subsequence 
K such that {x^}k x* . By passing to a subsequence if necessary, we can assume that 
h = I for all k e K. Let 

fi'' = Pk[Ax'-b-n,c4Ax'-b)]. 

We clearly see that 

{li'fn^,{Ax''-b) = 0. (62) 
Using (l^T]) and the definitions of $p and fi^, we have 

V/(x'=) + e -Afisjix'') + U{ek), Wk e /C. (63) 
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By ([58]), ([59]) and the definition of x^, 



one can observe that 




1 



g{x'';pk,Ik)\\ < Cfc- 



(64) 



L 



'Pk 



In addition, notice that Wfi'^W = pkdic*{Ax^ — b), which together with ( 159]) imphes that <t 
for all k. Hence, {p''} is bounded. By passing to a subsequence if necessary, we can assume 
that {fi^}K — ^ P*- Using (j64]l and upon taking limits on both sides of ( l62]) and (l63]l as 
/c e i^' — )■ cxD, we have 



In addition, since = for k ^ K, we know that = 0. Also, it follows from fl59]) that 
(i/c*(^3;* — 6) =0, which implies that Ax* — b & K,*. These relations yield that 



5 Concluding remarks 

In this paper we studied iterative hard thresholding (IHT) methods for solving /q regularized 
convex cone programming problems. In particular, we first proposed an IHT method and its 
variant for solving Iq regularized box constrained convex programming. We showed that the 
sequence generated by these methods converges to a local minimizer. Also, we established 
the iteration complexity of the IHT method for finding an e-local-optimal solution. We then 
proposed a method for solving /q regularized convex cone programming by applying the IHT 
method to its quadratic penalty relaxation and established its iteration complexity for finding 
an e-approximate local minimizer. Finally, we proposed a variant of this method in which the 
associated penalty parameter is dynamically updated, and showed that every accumulation 
point is a local minimizer of the problem. 

Some of the methods studied in this paper can be extended to solve some Iq regularized 
nonconvex optimization problems. For example, the IHT method and its variant can be 
applied to problem f[T2]) in which / is nonconvex and V/ is Lipschitz continuous. In addition, 
the numerical study of the IHT methods will be presented in the working paper [7j. Finally, it 
would be interesting to extend the methods of this paper to solve rank minimization problems 
and compare them with the methods studied in [SI [S] • This is left as a future research. 
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ip*fU,c*iAx* - 6) = 0, V/(x*) + e -A/6,(a:*) 



X* G Arg min{/(x) : Ax — 6 G /C*} 



and hence, x* is a local minimizer of ([2]). 
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